Smoothing piecewise linear activation functions based on mollified square root functions

Activation functions(AFs) are crucial components in neural networks for deep learning. Piecewise linear functions(PLFs) have been widely employed to act as AFs, thanks their computational efficiencies and simplicities. However, PLFs not completely differentiable, potential problems of training. The analytical expressions AFs based on pure can be smoothed via mollified square root function(MSRF) method, inspired by SquarePlus method ReLU approximation. In this paper, we propose a proposition defining maximum or minimum two PLFs, transform the results into function MSRF method. Based MSRF, modify well-known composed three four regularized ones systematically, including ReLU, LReLU, vReLU, Step, Biplolar, BReLU(Bounded ReLU), Htanh(Hard Tanh), Pan(Frying pan function), STF(Soft Thresholding Formulas), HTF(Soft SReLU(S-shaped MReLU(Mexican hat type TSF(Trapezoid-shaped function) functions. Additionally, according equivalences SoftPlus functions, some classic compound such ELU, Swish, Mish, SoftSign Logish DLU expressed also. derivatives versions demonstrate smoothness properties. proposed extended multiple easily, which will investigated deeply future.